Minimax rates of estimation for high-dimensional linear regression over $\ell_q$-balls

Consider the standard linear regression model Y = Xβ+w, where Y ∈ R is an observation vector, X ∈ R is a design matrix, β ∈ R is the unknown regression vector, and w ∼ N (0, σI) is additive Gaussian noise. This paper studies the minimax rates of convergence for estimation of β for lp-losses and in the l2-prediction loss, assuming that β belongs to an lq-ball Bq(Rq) for some q ∈ [0, 1]. We show that under suitable regularity conditions on the design matrix X , the minimax error in l2-loss and l2-prediction loss scales as Rq ( log d n )1− q 2 . In addition, we provide lower bounds on minimax risks in lp-norms, for all p ∈ [1,+∞], p 6= q. Our proofs of the lower bounds are information-theoretic in nature, based on Fano’s inequality and results on the metric entropy of the balls Bq(Rq), whereas our proofs of the upper bounds are direct and constructive, involving direct analysis of least-squares over lq-balls. For the special case q = 0, a comparison with l2-risks achieved by computationally efficient l1-relaxations reveals that although such methods can achieve the minimax rates up to constant factors, they require slightly stronger assumptions on the design matrix X than algorithms involving least-squares over the l0-ball.